CI: test Kubernetes port forwarder #4118
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Signed-off-by: Akihiro Suda <[email protected]>
I actually hadn't looked at this before, but I know that kind is hacking the preflight. --skip-phases=preflight Apparently always having 2 vCPU and 2 GiB total RAM is not so important, after all... But better to do a more selective ignore. |
|
The default CPU and memory seems enough to pass the CI |
| # Wish we could use BATS_TEST_RETRIES=3 as an environment variable here, | ||
| # but bats does not seem to support it. | ||
| uses: nick-fields/retry@ce71cc2ab81d554ebbe88c79ab5975992d79ba08 # v3.0.2 | ||
| with: | ||
| timeout_minutes: 30 | ||
| retry_on: error | ||
| max_attempts: 3 |
There was a problem hiding this comment.
You have to set the variable inside the test that is flaky; the harness sets it to 0, so you can't set it from the outside:
❯ cat random.bats
@test "flaky test that fails 70% of the time" {
BATS_TEST_RETRIES=10
echo "Attempt $BATS_TEST_TRY_NUMBER/$BATS_TEST_RETRIES" >&3
(( RANDOM % 10 < 3))
}
❯ ./hack/bats/lib/bats-core/bin/bats random.bats
random.bats
flaky test that fails 70% of the time 1/1
Attempt 1/10
✓ flaky test that fails 70% of the time
Attempt 2/10
1 test, 0 failures
❯ ./hack/bats/lib/bats-core/bin/bats random.bats
random.bats
flaky test that fails 70% of the time 1/1
Attempt 1/10
flaky test that fails 70% of the time 1/1
Attempt 2/10
flaky test that fails 70% of the time 1/1
Attempt 3/10
flaky test that fails 70% of the time 1/1
Attempt 4/10
flaky test that fails 70% of the time 1/1
Attempt 5/10
✓ flaky test that fails 70% of the time
Attempt 6/10
1 test, 0 failuresWithout the logging you cannot tell if the test needed retries to pass (and BATS_TEST_RETRY_NUMBER is not defined in setup or teardown).
There was a problem hiding this comment.
You can set it at the file level, so it applies to every test in the same file:
BATS_TEST_RETRIES=10
@test "flaky test that fails 70% of the time" {
echo "Attempt $BATS_TEST_TRY_NUMBER/$BATS_TEST_RETRIES" >&3
(( RANDOM % 10 < 3))
}There was a problem hiding this comment.
I feel GHA YAML is the right place to set BATS_TEST_RETRIES. Maybe we should submit a proposal to the upstream.
Can we just merge this PR?
| for tag in "${BATS_TEST_TAGS[@]}"; do | ||
| if [[ $tag =~ ^nodes:([0-9]+)$ ]]; then | ||
| nodes="${BASH_REMATCH[1]}" | ||
| fi | ||
| done |
There was a problem hiding this comment.
I'm not convinced that this is a good way to control setup for individual tests.
I think it would be clearer not to rely on local_setup, but to have each test call a setup function with an argument, like
@test 'Single-node configuration' {
setup_nodes 1
# ...
}
@test 'Multi-node configuration' {
setup_nodes "${LIMA_BATS_K8S_NODES:-3}"
# ...
}And local_teardown could use limactl ls --quiet | grep "^${NAME}-" to get the list of instances to delete.
This also allows you to control the number of nodes from the outside without modifying the test source code to adjust the tag.
(Ab)using the tags to pass arguments to the setup/teardown functions is a bit confusing, and should be simpler. But we can always change this later, e.g. when there actually is a multi-node test.
| done | ||
| } | ||
|
|
||
| k() { |
There was a problem hiding this comment.
I would call it kubectl for readability. Local functions always take precedence over running external programs.
3 participants
No description provided.